Domain adaption

ABSTRACT

The present invention relates to a method and system that allows input mammography images to be converted between domains. More particularly, the present invention relates to converting mammography images from the image style common to one manufacturer of imaging equipment to the image style common to another manufacturer of imaging equipment. Aspects and/or embodiments seek to provide a method of converting input images from the format output by one imaging device into the format normally output by another imaging device. The imaging devices may differ in their manufacturer, model or configuration such that they produce different styles of image, even if presented with the same raw input data, due to the image processing used in the imaging device(s).

FIELD

The present invention relates to a method and system that allows inputmammography images to be converted between domains, i.e. performingdomain adaption in mammography. More particularly, the present inventionrelates to converting mammography images from the image style common toone manufacturer of imaging equipment to the image style common toanother manufacturer of imaging equipment using Generative AdversarialNetworks.

BACKGROUND

Mammography is a medical imaging modality widely used for breast cancerdetection. Mammography makes use of “soft” X-rays to produce detailedimages of the internal structure of the human breast. These images arecalled mammograms and this method is considered to be the gold standardin early detection of breast abnormalities which provide a validdiagnosis of a cancer in a curable phase.

Unfortunately, the procedure of analysing mammograms is oftenchallenging. The density and tissue type of the breasts are highlyvaried and in turn present a high variety of visual features due topatient genetics. These background visual patterns can obscure theoften-tiny signs of malignancies which may then be easily overlooked bythe human eye. Thus, the analyses of mammograms even by highly skilledhuman operators, typically specialist doctors, often lead tofalse-positive or false-negative diagnostic results which may causemissed treatment (in the case of false-negatives) as well as unwantedpsychological and sub-optimal downstream diagnostic and treatmentconsequences (in the case of false-positives).

Most developed countries maintain a population-wide screening program, acomprehensive system for calling in women of a certain age group, freeof symptoms to have regular breast screening. These screening programsrequire highly standardized protocols to be followed by experiencedspecialist trained doctors who can reliably analyse a large number ofmammograms routinely. Most professional guidelines strongly suggestreading of each mammogram by two equally expert radiologists(industrially known as double-reading). Nowadays, with the number ofavailable highly skilled radiologists scarce and decreasing, thedouble-reading requirement is often impractical or impossible.

As such, computer aided diagnosis is becoming more popular as techniquesimprove. Typically, the current state of the art techniques for computeraided diagnosis use machine learning to provide the high level ofreliability required to be deployed in a clinical setting. To achievethis level of reliability, the machine learning algorithms need to betrained on data sets that allow reliable operation. However, there aremany manufacturers of imaging devices and so it is necessary to trainmodels/algorithms on training data from each imaging device with whichthe trained algorithm is to be used, but this isn't always possible andnew imaging devices are being developed and launched commercially thusit is an ongoing problem that models/algorithms need to be trained ontraining data using images for each existing and new imaging device inorder to be reliable.

Training models/algorithms in one domain (i.e. on a set of image datasourced from one manufacturer's imaging device) and then using thesetrained models/algorithms in another domain (i.e. on image data fromanother manufacturer's imaging device) can result in themodels/algorithms not performing well in the other domain.

SUMMARY OF THE INVENTION

Aspects and/or embodiments seek to provide a method of converting inputimages from the format output by one imaging device into the formatnormally output by another imaging device. The imaging devices maydiffer in their manufacturer, model or configuration such that theyproduce different styles of image, even if presented with the same rawinput data, due to the image processing used in the imaging device(s).

According to a first aspect, there is provided a computer-aided methodof training a neural network to transfer mammography images betweendomains, the neural network operable to perform the steps of: receivinga plurality of mammogram images in a first domain; receiving a pluralityof mammogram images in a second domain; determining a first network totransfer one or more of the plurality of mammogram images in a firstdomain to a second domain to output transferred second domain images;determining a second network to transfer one or more of the plurality ofmammogram images in a second domain to a first domain to outputtransferred first domain images; determining a discriminator network tooutput labels for each input image, the labels comprising being in thefirst domain, being in the second domain or being a generated image;wherein the training of the neural network is performed by optimising asum of losses.

By training a network constructed in this way, it is possible to train anetwork to transfer input mammography images between a first and seconddomain.

Optionally, each of the first and/or second networks to transfer one ormore of the plurality of images between domains comprises a pairedencoder and decoder

By using a paired encoder and decoder, the encoder can learn an abstractrepresentation of the features of the input image and the decoder canlearn how to translate the abstract representation of fractures into adomain specific image.

Optionally, optimising a sum of losses comprises optimising a sum oflosses between at least a plurality of: each of the plurality ofmammogram images in a first domain; each of the plurality of mammogramimages in a second domain, each of the labels; each of the outputtransferred second domain images; and each of the output transferredsecond domain images.

Optionally, the neural network is operable to receive at least onefurther plurality of mammogram images in one or more further domains andat least one further network to transfer one or more of the plurality ofmammogram images in one of the domains to another of the domains tooutput transferred another of the domains' images.

More than a first and second domain can be trained using this structureof network and approach, so third, fourth etc. domains can be trained.

Optionally, each of the losses making up the sum of the losses isweighted relative to each other of the losses making up the sum of thelosses.

By applying weights to each of the losses that make up the sum oflosses, the training process and/or network(s) can be fine-tuned.

Optionally, the training of the neural network is first done using lowresolution image data then iteratively in one or more steps theresolution of the image data is increased as the neural network istrained.

By using progressively growing techniques, the network can be trainedmore efficiently or effectively.

Optionally, the losses making up the sum of the losses includes anycombination of: a breast mask loss; a consistency loss; a reconstructionloss; a vendor loss; and a GAN loss.

Optionally, any of the first and/or second networks to transfer one ofmore of the plurality of images between domains comprises a generativenetwork.

Optionally, the first domain is a first vendor image style and thesecond domain is a second vendor image style.

According to a second aspect, there is provided a trained network totransfer one or more of the plurality of mammogram images in one domainto another domain, trained according to the method of any precedingclaim.

By outputting a trained network, optionally a trained encoder-decoderpair, the trained network can be used to transfer images betweendomains, for example from one manufacturer image format to another.

BRIEF DESCRIPTION OF DRAWINGS

Embodiments will now be described, by way of example only and withreference to the accompanying drawings having like-reference numerals,in which:

FIG. 1 illustrates an example flowchart showing a typical method used toperform mammography;

FIGS. 2 to 5 illustrate a set of input images in the style of the outputof devices from one imaging device manufacturer being converted usingthe method of the aspects/embodiments into the style of the output ofdevices from another imaging device manufacturer;

FIG. 6 illustrates an implementation of a generative adversarial networkused in aspects or embodiments; and

FIG. 7 illustrates an implementation of the method according toaspects/embodiment(s) described in more detail below.

SPECIFIC DESCRIPTION

FIG. 7 depicts an example embodiment of the present invention, whichwill be described below in more detail.

Referring first to FIG. 1, there is shown a typical process foranalysing and/or diagnosing a patient, or patients, using themammography process. Image data 110 is gathered using an imaging device(not shown). The imaging device is a “soft” x-ray device. The imagingdevices can be from multiple manufacturers, and each manufacturer has aproprietary method/software/process/algorithm 120 for converting the rawinput image data 110 gathered by the x-ray sensor into image data 130that can be understood by a human operator. Due to the proprietarynature of this process 120 for producing the image data 130, the outputimage data 130 varies subtly between imaging devices. Also, as over timemanufacturers may change their proprietarymethod/software/process/algorithm 120, it is possible that an olderimaging device may differ slightly in the output image data 130 from anew version of that imaging device from the same manufacturer, oralternatively compared to a different model of imaging device from thesame manufacturer regardless of whether the original sensor image data110 being exactly the same between models and/or manufacturers.

It may be worth noting that the detectors/sensors that generate the rawinput image data 110 are to a great extent commoditised. The maindifference between imaging devices from different manufacturers is theproprietary post-processing software/method/algorithm 120 that convertsthe raw input image data 110 into the output image data 130. In someexamples it is possible for the human eye to determine the different“styles” of image output from imaging devices from differentmanufacturers.

The output image data 130 is usually stored in a PACS (Picture Archiving& Communications System) 140, which is a data server for patient dataand in particular medical imaging data which is typically stored in theDICOM format.

The output image data 130 is either then sent for review 150, 160directly, or is at a later point in time extracted from the PACS 140 forreview 150,160. Typically, each review 150, 160 is carried out by anindependent trained specialist medical professional, a radiologist, whohas been trained to identify potentially malignant features/portions inthe patient images 130. This provides two independent review of eachcase, i.e. image data 130 associated with each patient. The PACS 140 maysometimes also provide historical image data 130 related to each case,i.e. from previous scans for each patient, either automatically or onrequest.

In some cases, the second review 160 is carried out by an automatedsystem. The automated system typically is a machine learned processusing neural networks trained on sample data from imaging devices.

Each of the reviews 150, 160 output a respective decision 170 a, 170 b(and optionally some metadata relevant to the decision, e.g. ifmalignancy is diagnosed an area of one or more images supporting thisdiagnosis may be annotated or highlighted). The combination of thesedecisions 180 is handled in different ways depending on the risktolerance of the respective medical professionals, but in the majorityof cases if either decision 170 a, 170 b indicates that there is adiagnosis of malignancy then the final decision 190 is that malignancyis present in the patient and further medical interventions are needed.

Referring now to FIG. 6, Generative Adversarial Networks (or GANs) willnow be briefly described.

In this example, the GAN will deal with images. Random noise 610 isgenerated as an input into the system shown in FIG. 6. The random noise610 is input into a generator network 620, which outputs fake images630. The fake images 630 are fed into a discriminator network 660 inparallel 650 with some real images 640. The discriminator network 660outputs whether it labels 670 each input image as real or fake.

The generator network 620, or generator, tries to predict features for agiven input. It functions to generate new data, in this example fakeimage data. The discriminator network 660, or discriminator, tries topredict a label (e.g. real or fake) for each input. The goal of trainingthe system of FIG. 6 is for the generator 620 to produce fake imagesthat are recognised as authentic/real by the discriminator 660.

To train the system, the discriminator 660 is in a feedback loop thatcan assess its labels against the ground truth of whether each inputimage 650 was real or fake and the label 670 output by the discriminator660 for each input image 650. Further, the generator 620 is in afeedback loop with the discriminator 660 so that it receives data onwhether the discriminator 660 output a real or fake label 670 for theinput image 650. The system as a whole is trained together, sometimes byallowing each of the generator 620 and discriminator 660 to learnseparately, and optionally by pre-training the discriminator 660 beforetraining the generator 620, and optionally by freezing thevalues/weights on each of the discriminator 660 or generator 620 whiletraining the other.

A more detailed introduction to GANs, which is hereby incorporated byreference, can be found athttps://towardsdatascience.com/generative-adversarial-networks-gans-a-beginners-guide-5b38eceece24.Further implementations of GANs are described in the following papers,which are also hereby incorporated by reference:

-   -   the “ StarGAN” approach: https://arxiv.org/pdf/1711.09020.pdf;    -   the “CycleGAN” approach: https://arxiv.org/pdf/1703.10593.pdf;        and    -   the “PGGAN” approach: https://arxiv.org/pdf/1710.10196.pdf.

Referring to FIG. 7, a specific embodiment will now be described.

One or more first domain images 715, e.g. images from an imaging devicefrom a first manufacturer, is taken as one input to the network of FIG.7. Each first domain image 715 is fed into a first paired encoder 725and decoder 735, which output a fake second domain image 745 where thedomain of the first domain image has attempted to be transferred to asecond domain. A loss is calculated between each first domain image 715and each respective fake second domain image 745 called the breast maskloss 751.

The fake second domain image 745 is then fed back into a second pairedencoder 730 and decoder 720 to output a fake first domain image 705. Theweights of the first and second paired encoders 725, 730 and decoders735, 720 are the same. A loss termed the reconstruction loss 710 iscalculated between each first domain image 715 and each respective fakefirst domain image 705 reconstructed from the fake second domain image745.

In alternative embodiments further domains may be included, requiringfurther encoder-decoder pairs, and these encoder-decoder pairs maintainthe same weights as the first and second paired encoders 725, 730 anddecoders 735, 720.

Each fake second domain image 745 and one or more second domain images740 are fed into a discriminator 750. A loss termed the consistency loss755 is calculated between each fake second domain image 745 and eachfirst domain image 715. The discriminator 750 labels 765 each inputimage as either in the first or second domains or a fake image. The lossbetween the labels for the first or second domains is calculated as thevendor loss 770 and the loss between the labels for the first or seconddomains and the label that one or more images are fake is calculated asthe GAN loss 760.

The total loss is a weighted sum of the breast mask loss 751, thereconstruction loss 710, the filter loss 755, the vendor loss 770 andthe GAN loss 760.

The network in FIG. 7 is trained to converge on a solution thatoptimises the total loss. More specifically, the training is done in aprogressively growing fashion, that is to say that the training startswith images of a very low resolution until the network has been trained.Then successive extra layers are added to the network, eachincrementally increasing the resolution of the images involved, untilthe network has been trained to handle the full desired resolution to beprocessed.

The network described above in effect learns the features that arevendor specific so that it can manipulate these features and transferacross domains. The result is an image-to-image or domain-to-domaintranslation model where one can input, for example, a first vendor imageinto the relevant trained encoder-decoder pair and the network outputsan image of the same mammogram, but it now looks like a second vendorimage (or any other manufacturer on which the network has been trained).Thus, the purpose of the network is to keep the key content the same butchange the “vendor features”.

If one has access to the raw detector data, transferring between imagedomains would be simple. If one has access to a referenced set of afirst vendor image, a second vendor image and a raw detector image ofthe same breast, one could learn the mapping function from vendor A toraw image to vendor B. However, it is not possible to have access to theraw detector images nor the proprietary algorithms/processing techniquesused by each vendor, nor is there any known sufficiently large datasetcontaining images from multiple vendors for the same patient. Insteadthe approach described in the specific embodiment can substantiallyautomatically learn the vendor features from a large dataset of images.

Referring now to FIGS. 2 to 5, examples of the use of the specificembodiment to adapt the domains of the imaging data 130 of patients isshown.

Specifically, in FIG. 2, the four mammography images 210 shown on theleft-hand side are real images taken with a second vendor imaging deviceand the images 220 shown on the right-hand side are the domaintransferred images having used the method of the specific embodiment totransfer the domain from a second vendor image style to a generatedfirst vendor style of image.

Then, in FIG. 3, the four mammography images 310 shown on the left-handside are real images taken with a first vendor imaging device and theimages 320 shown on the right-hand side are the domain transferredimages having used the method of the specific embodiment to transfer thedomain from a first vendor image style to a generated second vendorstyle of image.

Next, in FIG. 4, the four mammography images 410 shown on the left-handside are real images taken with a second vendor imaging device and theimages 420 shown on the right-hand side are the domain transferredimages having used the method of the specific embodiment to transfer thedomain from a second vendor image style to a generated first vendorstyle of image.

Finally, in FIG. 5, the four mammography images 510 shown on theleft-hand side are real images taken with a first vendor imaging deviceand the images 520 shown on the right-hand side are the domaintransferred images having used the method of the specific embodiment totransfer the domain from a first vendor image style to a generatedsecond vendor style of image.

As it is not possible to ascertain how each manufacturer processes 120the raw input data 110 into an output image 130, the method of thespecific embodiment allows for domain transfer without knowledge of anymanufacturer processing 120. This therefore allows training of machinelearned algorithms/models on image data 130 from only one manufacturer'simaging data to be used with imaging data from otherdomains/manufacturer's output images 130 as the input images 130 canundergo a domain transformation step prior to analysis by the machinelearned algorithm/model.

Machine learning is the field of study where a computer or computerslearn to perform classes of tasks using the feedback generated from theexperience or data gathered that the machine learning process acquiresduring computer performance of those tasks.

Typically, machine learning can be broadly classed as supervised andunsupervised approaches, although there are particular approaches suchas reinforcement learning and semi-supervised learning which havespecial rules, techniques and/or approaches. Supervised machine learningis concerned with a computer learning one or more rules or functions tomap between example inputs and desired outputs as predetermined by anoperator or programmer, usually where a data set containing the inputsis labelled.

Unsupervised learning is concerned with determining a structure forinput data, for example when performing pattern recognition, andtypically uses unlabelled data sets. Reinforcement learning is concernedwith enabling a computer or computers to interact with a dynamicenvironment, for example when playing a game or driving a vehicle.

Various hybrids of these categories are possible, such as“semi-supervised” machine learning where a training data set has onlybeen partially labelled. For unsupervised machine learning, there is arange of possible applications such as, for example, the application ofcomputer vision techniques to image processing or video enhancement.Unsupervised machine learning is typically applied to solve problemswhere an unknown data structure might be present in the data. As thedata is unlabelled, the machine learning process is required to operateto identify implicit relationships between the data for example byderiving a clustering metric based on internally derived information.For example, an unsupervised learning technique can be used to reducethe dimensionality of a data set and attempt to identify and modelrelationships between clusters in the data set, and can for examplegenerate measures of cluster membership or identify hubs or nodes in orbetween clusters (for example using a technique referred to as weightedcorrelation network analysis, which can be applied to high-dimensionaldata sets, or using k-means clustering to cluster data by a measure ofthe Euclidean distance between each datum).

Semi-supervised learning is typically applied to solve problems wherethere is a partially labelled data set, for example where only a subsetof the data is labelled. Semi-supervised machine learning makes use ofexternally provided labels and objective functions as well as anyimplicit data relationships. When initially configuring a machinelearning system, particularly when using a supervised machine learningapproach, the machine learning algorithm can be provided with sometraining data or a set of training examples, in which each example istypically a pair of an input signal/vector and a desired output value,label (or classification) or signal. The machine learning algorithmanalyses the training data and produces a generalised function that canbe used with unseen data sets to produce desired output values orsignals for the unseen input vectors/signals. The user needs to decidewhat type of data is to be used as the training data, and to prepare arepresentative real-world set of data. The user must however take careto ensure that the training data contains enough information toaccurately predict desired output values without providing too manyfeatures (which can result in too many dimensions being considered bythe machine learning process during training, and could also mean thatthe machine learning process does not converge to good solutions for allor specific examples). The user must also determine the desiredstructure of the learned or generalised function, for example whether touse support vector machines or decision trees.

The use of unsupervised or semi-supervised machine learning approachesare sometimes used when labelled data is not readily available, or wherethe system generates new labelled data from unknown data given someinitial seed labels.

Machine learning may be performed through the use of one or more of: anon-linear hierarchical algorithm; neural network; convolutional neuralnetwork; recurrent neural network; long short-term memory network;multi-dimensional convolutional network; a memory network; fullyconvolutional network or a gated recurrent network allows a flexibleapproach when generating the predicted block of visual data. The use ofan algorithm with a memory unit such as a long short-term memory network(LSTM), a memory network or a gated recurrent network can keep the stateof the predicted blocks from motion compensation processes performed onthe same original input frame. The use of these networks can improvecomputational efficiency and also improve temporal consistency in themotion compensation process across a number of frames, as the algorithmmaintains some sort of state or memory of the changes in motion. Thiscan additionally result in a reduction of error rates.

Developing a machine learning system typically consists of two stages:(1) training and (2) production. During the training the parameters ofthe machine learning model are iteratively changed to optimise aparticular learning objective, known as the objective function or theloss. An example in neural network training would be the backpropagationalgorithm, which is used in the described embodiment to train the model.Once the model is trained, it can be used in production, where the modeltakes in an input and produces an output using the trained parameters.

Any system feature as described herein may also be provided as a methodfeature, and vice versa. As used herein, means plus function featuresmay be expressed alternatively in terms of their correspondingstructure.

Any feature in one aspect may be applied to other aspects, in anyappropriate combination. In particular, method aspects may be applied tosystem aspects, and vice versa. Furthermore, any, some and/or allfeatures in one aspect can be applied to any, some and/or all featuresin any other aspect, in any appropriate combination.

It should also be appreciated that particular combinations of thevarious features described and defined in any aspects can be implementedand/or supplied and/or used independently.

1. A computer-aided method of training a neural network to transfermammography images between domains, the neural network operable toperform the steps of: receiving a plurality of mammogram images in afirst domain where the first domain is a first vendor image style;receiving a plurality of mammogram images in a second domain wherein thesecond domain is a second vendor image style; determining a firstnetwork to transfer one or more of the plurality of mammogram images ina first domain to a second domain to output transferred second domainimages; determining a second network to transfer one or more of theplurality of mammogram images in a second domain to a first domain tooutput transferred first domain images; and determining a discriminatornetwork to output labels for each input image, the labels comprisingbeing in the first domain, being in the second domain, being thetransferred first images or being the transferred second images; whereinthe training of the neural network is performed by optimising a sum oflosses.
 2. The method of claim 1 wherein each of the first and/or secondnetworks to transfer one or more of the plurality of images betweendomains comprises a paired encoder and decoder.
 3. The method of claim 1wherein said optimising a sum of losses comprises optimising a sum oflosses between at least a plurality of: each of the plurality ofmammogram images in a first domain; each of the plurality of mammogramimages in a second domain, each of the labels; each of the outputtransferred second domain images; and each of the output transferredsecond domain images.
 4. The method of claim 1 wherein the neuralnetwork is operable to receive at least one further plurality ofmammogram images in one or more further domains, wherein the one or morefurther domains comprise one or more vendor image styles, and at leastone further network to transfer one or more of the plurality ofmammogram images in one of the first or second domains domains to theone or more further domains to output transferred one or more furtherdomains' images.
 5. The method of claim 1 wherein the each of the lossesmaking up the sum of the losses is weighted relative to each other ofthe losses making up the sum of the losses.
 6. The method of claim 1where the training of the neural network is first done using lowresolution image data then iteratively in one or more steps theresolution of the image data is increased as the neural network istrained.
 7. The method of claim 1 wherein the losses making up the sumof the losses includes any combination of: a loss calculated between theone or more of the plurality of mammogram images in a first domain andeach of the transferred second images; a loss calculated between each ofthe transferred second images and the one or more plurality of mammogramimages in a second domain; a loss calculated between the one or more ofthe plurality of mammogram images in a first domain and each of thetransferred first images; a loss calculated between the labels for thefirst or second domains; and a loss calculated between the labels forthe first or second domains and the labels for transferred first imagesor transferred second images.
 8. The method of claim 1 wherein any ofthe first and/or second networks to transfer one of more of theplurality of images between domains comprises a generative network. 9.(canceled)
 10. A trained network to transfer one or more of theplurality of mammogram images in one domain to another domain, trainedaccording to the method of claim
 1. 11. The method of claim 1 furthercomprising learning features associated to each domain.
 12. The methodof claim 11, wherein the features associated are operable to bemanipulated and transferred across domains.